Skip to content

Feat/content safety decision - #1196

Merged
yaozheng-fang merged 14 commits into
mainfrom
feat/content-safety-decision
Nov 5, 2026
Merged

Feat/content safety decision#1196
yaozheng-fang merged 14 commits into
mainfrom
feat/content-safety-decision

Conversation

@wujiaming-ai

@wujiaming-ai wujiaming-ai commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@wujiaming-ai
wujiaming-ai force-pushed the feat/content-safety-decision branch from 3e0e8bf to 4dbf6cf Compare November 3, 2026 06:51
wujiaming-ai and others added 12 commits November 4, 2026 15:25
@wujiaming-ai @claude
feat(content-safety): add opt-in decision API auditing
bdcb07e 
Add an opt-in content safety plugin that audits content at the model and
tool lifecycle checkpoints through the System One decision API. The default
provider remains none and nothing is mounted automatically.

- strict tool.content_safety configuration with Ark connection defaults
- strict decision response arbitration and on_error policy
- streaming partial suppression with final aggregated audit
- audited SkillToolset load/resource/script/list/search tools
- LLM Shield adapter and real-ADK E2E coverage

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
test(content-safety): add opt-in live System One integration test
afcc952 
Exercise the real Ark System One endpoint through the production plugin
path when VEADK_CONTENT_SAFETY_LIVE=1 and a key are supplied. Normal test
runs skip it so no CI job calls the production service by accident.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
test(content-safety): avoid duplicate pytest test basenames
086f6c0 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
refactor(content-safety): rename decision plugin and drop skill adapter
c554b49 
Rename ContentSafetyPlugin to DecisionPlugin to match the reviewed
architecture, remove the skill pre-audit adapter from this change, and
use StateBuilder.build_after_model for model-output auditing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
fix(content-safety): reject unsafe non-Ark decision fallback
aede39b 
Only derive the System One endpoint from a known Ark HTTPS base. Require
an explicit decision.api_key for custom endpoints so Ark credentials are
never silently sent to another provider.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
fix(content-safety): ignore hidden thought parts in model audits
d91b792 
ADK may attach hidden reasoning as Part(thought=True). Auditing that text
caused benign final answers to be blocked after the model had already
produced safe visible output. Filter hidden thought parts from after_model
state and preserve them when replacing visible event text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
fix(content-safety): apply llm_shield app id before legacy import
c1ca22c 
The legacy LLM Shield module constructs a plugin at import time, so a
config-only llm_shield_app_id was read too late and startup failed unless
TOOL_LLM_SHIELD_APP_ID was already exported. Set the env inside the lock
before importing, and restore it even when the import raises.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
refactor(decision): extract reusable Decision API client
c9a9eb9 
Move the Decision API protocol into veadk.decision with typed request and
response models. content_safety now adapts its question configuration at the
boundary and consumes the typed response, keeping the dependency one-way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
fix(content-safety): harden decision and llm shield paths
9c96c6f 
Prevent non-Ark credentials from being reused for an explicit Ark decision
endpoint. Audit visible text in mixed text/function-call events, run legacy
LLM Shield callbacks outside the event loop, load .env before applying
decision overrides, reject plaintext remote endpoints, and store API keys as
SecretStr. Add regression coverage for each fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
feat(content-safety): auto-mount from config.yaml
3fc483f 
Mount the content-safety plugin from VeADK Runner and AgentKit when
`tool.content_safety` is present, so callers no longer have to load the
config, build the plugin, and wire it into App manually.

- Agent gains an optional `content_safety` field (None/False/True/dict/config)
  and a `content_safety_plugins` property for manually built ADK Apps.
- `load_content_safety_config_if_present()` treats missing files or blocks as
  disabled while still failing fast on invalid configuration.
- Plugin resolution deduplicates by name and keeps explicitly supplied plugins.
- Runner now supports app-only construction while injecting plugins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
feat(content-safety): enable built-in risk questions by default
0257704 
- merge five built-in questions for provider=decision
- allow per-name overrides and appends, add use_default_questions
- fix question type to noul and reject user-supplied type

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
feat(content-safety): add stable policy prefix to audit state
e810148 
- prepend fixed audit policy outside the untrusted payload
- shorten built-in question instructions
- keep truncation limited to dynamic content

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai
wujiaming-ai force-pushed the feat/content-safety-decision branch from 59dfc5c to e810148 Compare November 4, 2026 07:27
wujiaming-ai and others added 2 commits November 4, 2026 17:00
@wujiaming-ai @claude
feat(content-safety): expand built-in risk questions to 18 categories
9ea8529 
- merge the documented 12 categories with existing defaults
- add self_harm, child_safety, and fraud_social_engineering
- keep existing names and the 0.5 default threshold

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wujiaming-ai @claude
feat(content-safety): tier default question thresholds by risk
6f44e30 
Lower thresholds for safety-critical categories (child safety and
self-harm) and raise them for broad, high-false-positive categories such
as compute abuse and public order. Export the per-question threshold
mapping and update tests to assert the tiering.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@yaozheng-fang
yaozheng-fang merged commit 9972543 into main Nov 5, 2026
35 of 39 checks passed
@cuericlee

cuericlee commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Code review: opt-in content safety plugin

Reviewed the full diff against the google-adk>=2.7.0,<2.8.0 contract (BasePlugin, PluginManager, runners.py, base_llm_flow.py, and both tool-execution paths in functions.py). Overall this is a well-structured, strictly opt-in feature, and the critical interception paths (streaming partials withheld until audited, before_tool short-circuiting real execution, on_event replacement before persistence/yield) are covered by real Runner e2e tests. Conditionally ready to merge; I'd suggest clarifying finding 1 first.

Suggested clarifications / fixes

  1. Multimodal content is never audited (scope gap). StateBuilder extracts only part.text and skips thoughts, so image/audio/video/file inputs at before_model and multimodal model outputs at after_model produce a skipped (allow) result. If text-only auditing is intentional for this PR, please state it explicitly in the module docs/README; otherwise document a follow-up strategy.

  2. Redaction is key-only. state._redact replaces values only by dictionary key name; secrets embedded in string values (e.g. logs like token=..., JSON strings, Authorization: Bearer ...) are sent to the decision API verbatim. Consider regex-based redaction for string values, or document that this is best-effort and not DLP.

  3. Delimiter injection in the defuse wrapper. Audited text is wrapped in <untrusted>...</untrusted> without escaping, so a literal </untrusted> in user input or a tool response can close the "untrusted" section early from the judge model's perspective. Escaping angle brackets (or using a random per-request delimiter) would close this.

  4. Connection setup per audit. DecisionClient.judge creates and tears down a new httpx.AsyncClient on every call, with up to four serial audits per turn (10s timeout each). Reusing a client across the plugin's lifetime (and optionally skipping/short-circuiting overlapping before_tool/after_tool audits) would avoid amplifying judge-service tail latency onto every agent request.

  5. after_tool cannot undo side effects. In ADK 2.7.1 the tool runs first; an after_tool dict only replaces the function response seen by the model, it does not roll back executed actions. before_tool is the real gate. Worth calling out in the docs so users don't over-trust the four-checkpoint framing.

  6. Minor: only _audits is bounded by _MAX_INVOCATIONS; _stream_buffers and _audited_text_hashes rely on after_run_callback always firing and could grow if a run is cancelled before cleanup — consider bounding them too.

  7. Minor: llm_shield_app_id is accepted (silently unused) when provider=decision; a cross-field validation or warning would help. Also consider documenting that only the last user message and latest tool response are included in the before_model audit window.

Notes

  • I did not execute the test suite locally (no google-adk 2.7.x environment); findings above come from static reading plus cross-checking the pinned ADK source.
  • Default-question quality/false-positive rate is a model-side concern outside this review; the opt-in live test is a good pre-release signal.

Happy to expand any item with file/line references.

@cuericlee cuericlee mentioned this pull request Nov 5, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaozheng-fang yaozheng-fang yaozheng-fang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wujiaming-ai @cuericlee @yaozheng-fang